Skip to content

Per-project test execution, and fork-death diagnostics that name a cause - #686

Open
oyvindberg wants to merge 18 commits into
masterfrom
test-execution-modes
Open

Per-project test execution, and fork-death diagnostics that name a cause#686
oyvindberg wants to merge 18 commits into
masterfrom
test-execution-modes

Conversation

@oyvindberg

@oyvindberg oyvindberg commented Sep 9, 2026

Copy link
Copy Markdown
Owner

Test running grows up: a real execution model, and forks that can't die in silence

This turns bleep test from "spawn a JVM per suite and hope" into a proper, configurable execution model — and, just as importantly, makes a forked test JVM incapable of dying without telling you why. Two halves: a small, well-documented config surface you'll actually reach for, and a big pile of hardening so a weird test never again shows up as a mysterious "server may have crashed."


Part 1 — how a project's tests run, in your control

Everything is driven by ordinary project fields. Here's the full surface, in one place:

projects:
  myapp-test:
    isTestProject: true            # this project is tests → suites get discovered & run
    dependsOn: myapp
    dependencies:
      - org.scalatest::scalatest:3.2.19

    # --- the two test-execution knobs (both optional) ---
    testFork: per-project          # per-project (default) | per-suite
    maxConcurrentSuites: 1         # how many suites at once (default 1 = sequential)

    # --- occasionally useful ---
    testFrameworks: []             # escape hatch: name a framework bleep can't auto-detect
    testTags:                      # tag → suite-name patterns, for --only-tag/--exclude-tag
      slow: ["*IntegrationSpec"]
    platform:
      jvmOptions: ["-Xmx2g"]       # JVM options for this project's test fork

Frameworks are auto-detected from the classpath — ScalaTest, JUnit 4/5, MUnit, utest, Specs2, ScalaCheck, weaver, ZIO Test, Kotest, Cucumber, jqwik and more. You usually write just isTestProject: true.

testForkwhere suites run, and what that costs you

per-project (default) per-suite
Model one forked JVM for the whole project (maven's forkCount=1 reuseForks=true) one forked JVM per suite, pooled by classpath
You get warm reuse, one live suite's heap at a time, and shared fixtures that survive across suites — a booted app, a Testcontainers instance, a schema an earlier suite created OS-level isolation: a suite that calls System.exit, leaks a thread, or corrupts a static takes down only its own process, which bleep replaces
You pay suites share a JVM, so a suite that hard-exits the JVM takes its siblings with it (now fully diagnosed — see Part 2) many JVM starts; no fixture reuse
Concurrency maxConcurrentSuites (JUnit Platform only) maxConcurrentSuites bounds how many forks run at once

Default is per-project + sequential. That's the memory-frugal, correct-for-everything choice; your cores stay busy because bleep runs many projects' forks at once under a machine-wide governor, not many suites inside one fork.

maxConcurrentSuiteshow many at once, and the one rule

In per-project mode this only speeds up JUnit Platform suites: the JUnit engine runs that many of the project's classes at once inside the one fork, each as its own launcher execution. bleep sets no JUnit configuration parameters, so a junit-platform.properties you ship still governs parallelism within a class (@ResourceLock/@Execution and all) — bleep neither enables nor overrides it. Because each class runs in a separate execution, @ResourceLock is not coordinated across classes, so keep maxConcurrentSuites at 1 when a project's classes share mutable state. That's deliberate — JUnit is the one runner whose ecosystem is built for parallel execution.

What is NOT allowed / has no effect:

  • sbt-interface frameworks never run concurrently inside a shared fork. They share one Runner, expose no conflict graph, and generally aren't written for it. Set maxConcurrentSuites > 1 on an sbt-only project and it's a no-op — bleep emits a BSP warning telling you so. To run sbt suites concurrently, switch to testFork: per-suite (a fork each, safely isolated).
  • So the honest matrix is: JUnit → concurrency via maxConcurrentSuites; sbt → concurrency via per-suite. One rule, and everything else follows.

Why per-project honors stateful frameworks

The sbt test interface's contract is one Framework/Runner per framework, done() called once per run. per-project runs a project's suites of a framework through exactly that — one Runner, one done(). Frameworks that keep run-level state or report a suite's detail from done() (weaver's <system-err>, hedgehog's <system-out>) depend on it; give each suite its own fresh Runner/done() and that reporting is lost. Frameworks that can't share a Runner at all are detected and given their own fork per suite automatically (weaver, hedgehog today). Result: the framework matrix goes from 6/18 → 18/18 green under per-project. JUnit Platform gets the analogous treatment — one launcher.execute per class on a single shared LauncherSession, so a session-scoped fixture builds once and per-class attribution stays exact.


Part 2 — a forked JVM can no longer die in silence ⭐

This is the part that will save you a debugging session. Previously, a test fork that died oddly surfaced as "BSP server connection lost (server may have crashed)" — which points at memory, when the real cause was a stale build, a bad -Xmx, or a test calling System.exit. Every one of those is now caught and named.

  • Abandoned forks exit instead of spinning forever. A fork whose parent went away used to loop on a broken socket at ~100% CPU, re-encoding the same error — one was found pegged for ~2 hours. These zombies pile up and starve the machine, which then kills fresh forks at startup: a self-inflicted cascade that looked like "N suites never reported a result." The command loop now exits on a broken/closed socket, bounded so no failure state can spin.

  • Death without finalizers is detected and distinguished. A test that calls Runtime.halt() — or a native exit() — kills the JVM running no shutdown hooks at all, so nothing it writes to stderr survives the pipe teardown. The fork now writes its exit diagnostic (did the command loop end on its own terms? which suites were still running? a full thread dump) to a file the parent hands it and reads after death. The file's absence is itself the signal: no file ⇒ no hook ran ⇒ it was Runtime.halt/a hard kill, not System.exit. That single bit ends the guessing.

  • System.exit names its caller. When a test ends the JVM the ordinary way, a shutdown hook dumps every thread to fd 2 — the one parked in java.lang.Shutdown/Runtime.exit is the culprit — and the parent folds that into the failure.

  • A failed fork's stderr is actually captured. We now drain an exited fork's streams to EOF instead of reading the zero bytes available() at the instant of death. That's what finally surfaces Unrecognized VM option …, Could not create the Java Virtual Machine, and hs_err pointers — the common startup failures that used to vanish.

  • "N suites never reported a result" became a lead, not a wall. A batch fork that stops mid-run now reports the drained stderr, whether the JVM is still alive (wedged) vs. exited, and a thread dump when wedged.

  • Every bleep-initiated kill is recorded. JvmPool.kill — the single chokepoint every eviction/timeout/cancellation/shutdown passes through — now emits an onForkKill event (pid, reason, was-it-alive), recorded on the fork-lifecycle stream and joinable to fork_start/fork_end by pid. So "did bleep end this fork, or did it exit itself?" is answerable after the fact: a fork_end with no preceding fork_kill was not bleep's doing.

  • Test discovery no longer kills the client over a stale .class. Reflecting over compiled classes can hit an orphaned .class left by an incremental compile after a package rename, whose supertype no longer exists → NoClassDefFoundError, a LinkageError that Try/NonFatal do not hold. It used to tear down the whole client. Now one un-reflectable class costs only that class: it's skipped, logged (with "bleep clean on that project clears it"), and discovery continues.

  • Diagnostics go through the right channel. The parent-side breadcrumbs are structured (Logger) or recorded as metrics events, not raw System.err — levels, context, and joinable records instead of loose lines.

And a floor under all of it: a broken build can no longer render as a green "0 tests passed", and a kill the user requested is reported as a cancellation, not a failure.

oyvindberg and others added 18 commits September 3, 2026 23:22
Make per-project (maven's forkCount=1 reuseForks=true) the default test JVM
mode, replacing per-suite. An unset testJvm now runs a project's suites in one
shared fork; testSuiteParallelism bounds concurrency inside it and defaults to
~cores/4 (per-suite still means a fork per suite, unbounded by default). The
machine-wide governor caps the total across projects either way.

The default is resolved in one place (testJvm.getOrElse(PerProject)) at the two
sites that read it, and suiteParallelism's default is now mode-aware so both the
JUnit-Platform batch path and the shared suite-by-suite path get ~cores/4 in
per-project mode.

Docs-driven: adds docs/usage/testing.mdx (the execution model, when to reach for
each mode, JVM options, selecting what runs), registers it and the existing
test-tags page in the sidebar, adds the missing testJvm field to schema.json,
and refreshes the model scaladoc and testSuiteParallelism's stale description.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
Genericize every Quarkus reference in the test-execution code, comments, docs,
and test data — the general "run a project's suites in one JVM" feature stands
on its own, described in terms of the mechanisms it uses (a singleton-per-JVM
application, a LauncherSessionListener, shutdown hooks that stop containers)
rather than one framework. The Quarkus-specific naming and rationale belong in
the later Quarkus PR, which is where `@QuarkusTest` actually enters the build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…form

The per-project shared-session support calls JUnitPlatformRunner.enableSharedSession()
(and closeSharedSession() in the finally) unconditionally at fork startup. Both
merely reference JUnitPlatformRunner, which links against org.junit.platform.launcher.*
— and junit-platform-launcher is a `provided`, compile-only dependency of
bleep-test-runner, so it is NOT on the runtime classpath of a fork for an sbt
test-interface project (ScalaTest, MUnit, utest, Specs2, ScalaCheck, weaver, ZIO
Test). Referencing the class there throws NoClassDefFoundError before the fork sends
Ready, so every one of those suites failed with "Expected Ready, got Exception in
thread main" — i.e. the whole feature broke the majority of Scala test projects.

Guard both calls with a lazy Class.forName probe for a launcher class: touch
JUnitPlatformRunner only when JUnit Platform is actually present. An sbt-interface
fork has no session to share and none to close, so skipping is correct, not a
fallback. JUnit-Platform forks are unaffected.

Also genericizes the two remaining Quarkus mentions in this file (they belong in
the later Quarkus PR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
… fork

Per-project mode now runs a project's suites of one framework through a single
execution — the sbt interface's one-runner-per-framework contract, maven's
forkCount=1 reuseForks=true. Before, a shared fork created a fresh Framework/Runner
(and called done()) per suite, which corrupted stateful frameworks (munit, ZIO,
hedgehog), and the JUnit "batch" ran one launcher.execute() over all classes, which
lost per-class attribution for engines whose test tree is not keyed by the selected
class (cucumber features, spek specs reported zero).

- SuiteRunner: runOneSuiteOn(framework, runner, ...) + runSuites(...) — one Runner
  for all a framework's suites, done() once, sequential by default with a degree knob.
- JUnitPlatformRunner.runSuites: one launcher.execute PER class on the shared
  LauncherSession (session-scoped fixtures — a @QuarkusTest app — still build once),
  restoring per-class attribution.
- Protocol RunSuites generalized to carry frameworkClass + args and accept
  sbt-test-interface; ForkedTestRunner routes JUnit vs sbt.
- DAG groups a project's JVM suites by framework into one batch each (JUnit and each
  sbt framework); PlatformRunner (JS/Native) suites stay per-suite. Default degree 1
  (sequential — the safe maven default); testSuiteParallelism opts into concurrency.
- Output attribution: a fork-level active-suite fallback tags output from a
  framework's own threads (ZIO fibers, specs2 workers) when the thread-local is unset,
  so a batched suite's <system-out> is not lost.

Matrix (JvmScala213TestFrameworkIT) goes 6/18 -> 16/18 under per-project; weaver and
hedgehog still drop a failing test's late output (follow-up).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…ehog)

A few sbt frameworks report a suite's failure detail from Runner.done() — called
once per run — or on their own threads after the suite's tasks return. Sharing one
Runner across a project's suites (one done(), per-project mode) drops that per-suite
output: weaver's <system-err> and hedgehog's <system-out> lost the failing test's
reason. Give such a framework its own fork per suite (own done()) regardless of the
project's mode, via a small class-keyed denylist (FrameworkSelection.needsIsolatedFork);
they are left out of batching and their suite tasks force an exclusive fork.

JvmScala213TestFrameworkIT is now 18/18 under per-project (every framework).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
Correct the testing guide, scaladoc, and schema now that per-project's default
degree is 1 — suites run one at a time in the shared fork, exactly maven surefire's
reuseForks=true, the safe default since not every framework's engine is re-entrant.
testSuiteParallelism > 1 opts into concurrency. Also fix the guide's now-stale
description of the JUnit path: it runs one launcher execution per class on a shared
LauncherSession (session-scoped fixtures build once, per-class attribution exact),
not a single execution over all classes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…Fork/maxConcurrentSuites

The safe default: in per-project mode, sbt-interface frameworks always run
their suites sequentially through one Runner/one done(). They share a Runner
and have no lock-aware scheduler, so concurrency in a shared JVM is unsafe
(shared statics, unattributable async output). maxConcurrentSuites raises
concurrency only for JUnit Platform, whose engine owns a lock-aware scheduler;
concurrency for sbt suites is testFork: per-suite (a fork each, OS-isolated).
Per-project default concurrency is now 1 (was ~cores/4) — memory-frugal, one
live suite's heap at a time; cores are saturated across projects by the
governor. An sbt-only project that sets maxConcurrentSuites > 1 gets a BSP
build/logMessage warning that it has no effect.

Renames (public YAML surface): testJvm -> testFork (values unchanged),
testSuiteParallelism -> maxConcurrentSuites; type TestJvmMode -> TestForkMode.

Cleanup: removed the now-dead concurrent (thread-pool) branch from
SuiteRunner.runSuites — sbt batches always run at degree 1 — and its degree
parameter. Also fixed two pre-existing stale DiscoveryResult(batchParallelism=)
references that no longer compiled (the field is `batches`).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ForkedTestRunner's command loop wrapped `in.readLine()` in the same catch as
command dispatch and, on any exception, looped straight back to reading. When
the parent (bleep) goes away — it gave up on the fork, or the client
disconnected — the protocol socket breaks and readLine() throws on every call,
so the fork spun at ~100% CPU forever, re-encoding the same error each
iteration. Observed in the wild: a fork stuck ~2h with `main` pegged in
TestProtocol.encodeError. These zombies accumulate, starve CPU/RAM, and then
freshly-spawned forks get killed at startup — a self-inflicted cascade that
looks like "N suites never reported a result" with no cause.

Read the socket in its own try: a read failure (or EOF) means the parent is
unreachable, so exit and let the finally block clean up. Per-command dispatch
errors are still reported and tolerated, now bounded by a consecutive-error cap
so no other persistently-failing state can spin either.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…it unit-testable

When a test fork never connects back, bleep already tried to quote what it wrote
— but read only the bytes available() at the instant the process was seen dead,
which is almost always zero: the JVM's "Unrecognized VM option ...", "Could not
create the Java Virtual Machine", or an hs_err pointer is flushed a beat later.
So the reason was dropped and the failure surfaced as a bare "N suites never
reported a result" — the common case (a bad JVM option, a startup crash) hidden
behind the mechanism meant for the uncommon weird one.

Split reading by whether the fork has exited: an exited fork's writer is closed,
so drain its streams to EOF (cannot block) and get the whole tail; a still-alive
fork is read non-blocking as before. Moved describeChildOutput / drainAvailable /
drainToEof to object JvmPool as private[testing] pure functions taking a bare
Process, so DescribeChildOutputTest exercises the real capture (java with a bad
option → its stderr) without standing up a pool or the BSP server.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a batched fork stopped after reporting some suites but not others, the
result was a bare "batched suites produced no result: <names>" — cause unknown,
a dead end. But a fork that dies mid-run of an OutOfMemoryError, a native crash,
or System.exit records the reason on its OWN stderr (fd 2), which the per-suite
protocol never carries. Drain that stderr into the failure, note whether the
fork is still alive (wedged) vs exited, and add a thread dump when it is wedged.
"N never reported a result" becomes a lead instead of a wall.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…exit

A test that calls System.exit()/Runtime.halt() ends the whole forked JVM; on
JDK 24+ (no SecurityManager) bleep cannot block it, and every suite that had not
reported is lost. The parent saw only a clean exit-0 and could do no better than
"likely System.exit()". Register a shutdown hook that fires only on such an
unrequested shutdown (the command loop had not exited on its own terms) and
writes a full thread dump to fd 2 — the thread parked in java.lang.Shutdown /
Runtime.exit is the caller. The parent already drains fd 2, and the batch paths
now append that (forkDeathDiagnostic) to the failure, so "N never reported"
finally names the culprit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gnostic, and every bleep-initiated kill is logged

A fork that exits from under a run — a test's System.exit, or a Runtime.halt that
skips shutdown hooks entirely — left the parent with a bare "exited 0" and no cause,
because whatever it wrote to stderr races the pipe teardown and is routinely lost.

- The fork now writes its exit diagnostic (whether the command loop had ended on its
  own terms, which suites were still running, a full thread dump) to a FILE the parent
  passes in and reads after death. A file survives the teardown that loses stderr, and
  its ABSENCE is itself a signal: no file means no shutdown hook ran — a Runtime.halt
  or hard kill, not a System.exit.
- The command loop records WHY it ended (EOF / IOException / Shutdown / error-cap) and,
  on teardown, names the suites still running that it is abandoning.
- JvmPool.kill — the single socket-close chokepoint — logs every bleep-initiated kill
  with pid, liveness and reason, so a fork's death that is NOT preceded by such a line
  is known to be self-inflicted rather than bleep's doing.
- describeExit no longer asserts "likely System.exit()" for a clean exit-0; it names
  the three possibilities and points at the exit log to disambiguate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…the client

Reflecting over a project's compiled classes (getDeclaredMethods / getAnnotations /
Class.isAssignableFrom, in both the fingerprint and annotation strategies) can land on
a class whose supertypes or annotations name a type no longer on the classpath — almost
always an orphaned .class an incremental compile left behind after a package rename.
That throws NoClassDefFoundError, a LinkageError, which is an Error, not an Exception —
so scala.util.Try / NonFatal do NOT hold it. It escaped discovery, tore down the whole
client handler, and surfaced to the user as "BSP server connection lost (server may have
crashed)", pointing at memory when the cause was a stale build.

One un-reflectable class now costs only that class: matchFingerprint and
detectFrameworkByAnnotation run under a guard that catches LinkageError (separately from
NonFatal, so OutOfMemoryError still propagates), logs which class was skipped and that
`bleep clean` on that project is the likely fix, and lets discovery continue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…per channels, not raw System.err

The fork-lifecycle and failure diagnostics added earlier printed straight to
System.err, which lands in the daemon log but without the level/context framing
the rest of the server logs with. Route each to where it belongs:

- Fork kills (the JvmPool.kill chokepoint) become a JvmPoolListener.onForkKill
  event, recorded by BspMetrics as a `fork_kill` entry — joined to fork_end/fork_start
  by pid, so "did bleep end this fork, or did it exit itself?" is answerable from the
  metrics stream (a fork_end with no preceding fork_kill was not bleep's doing) rather
  than by grepping raw stderr. Keeps bleep-core logger-free: the pool announces via the
  listener it already has, exactly as onForkStart/onForkEnd do.
- The batch fork-error / fork-diagnostic messages (TestRunner) and the stale-class
  discovery skip (ClasspathTestDiscovery) now go through ryddig's Logger — threaded in
  as a parameter — so they carry a level and read like every other server log line.

Behaviour is unchanged; only the destination and framing. The fork-side diagnostics
(fd 2 + the exit-log file) stay as they are — a separate process has no other channel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
The test-execution model added fields to the project model (testFork,
maxConcurrentSuites) and schema, which flow into the resolved build the snapshot
tests cache. Regenerate the caches so the snapshots reflect the current model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
This PR keeps Quarkus specifics out (they enter in the follow-up that wires
@QuarkusTest into the build), but the testing guide still used @QuarkusTest as the
session-scoped-fixture example — a reader cannot use it yet. Describe the fixture by
what it is (a booted application, a LauncherSessionListener), matching how the code and
its comments already phrase it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…perties

bleep sets no JUnit configuration parameters — the discovery request is
`request().selectors(selectClass).build()` — so a junit-platform.properties on the
classpath still governs parallelism WITHIN a class, and bleep neither enables nor
overrides it. maxConcurrentSuites is bleep's own thread pool running that many classes
at once, each as its OWN launcher.execute; a stale docstring claimed the opposite
("configuration parameters set here override any junit-platform.properties").

Also record the consequence the old text implied away: because each class runs in a
separate execution, @ResourceLock is not coordinated ACROSS classes — keep
maxConcurrentSuites at 1 when a project's classes share mutable state. (Within a class,
jupiter's own parallel execution does honor the locks, if you enable it via properties.)

And genericize the last @QuarkusTest mention in this file's comments (Quarkus stays out
of this PR).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant